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Introduction 


Disruptive student behavior is a common stressor for elemen- 
tary school teachers and can significantly detract from instruc- 
tion time (Greenberg et al. 2016; Robb et al. 2011). Although 
classroom management strategies and targeted interventions 
(e.g., daily report card) are effective in preventing and reducing 
disruptive behavior (Epstein et al. 2008; Pyle and Fabiano 
2017), teachers’ use of some of these strategies is limited 
(Hart et al. 2017; Owens et al. 2018). One means to support 
teachers’ use of interventions is via problem-solving consulta- 
tion with performance feedback (also referred to as coaching) to 
improve teacher knowledge about the interventions and skills in 
applying the interventions (Frank and Kratochwill 2014). Yet, 
even when teachers receive this support, intervention imple- 
mentation is variable (Fabiano et al. 2010; Owens et al. 
2019), suggesting that other factors (in addition to teacher 
knowledge and skills) may influence implementation (Han 
and Weiss 2005). One hypothesized variable is teacher beliefs 
(e.g., about the importance of the intervention or confidence in 
their own skills). Indeed, because implementation integrity of- 
ten drops precipitously when consultation is withdrawn 
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(Mortenson and Witt 1998; Noell et al. 1997), other factors such 
as teacher self-efficacy and motivation may be needed for on- 
going strategy use. Given that low and variable implementation 
can severely compromise student outcomes (Conroy et al. 
2015; Owens et al. 2017), innovative consultation methods that 
address knowledge, skills, and teacher beliefs may be needed to 
adequately support teachers who are struggling with implemen- 
tation integrity. 


Innovations in Consultation: The Use of Motivational 
Interviewing 


In response to this need, our team and others have developed 
and evaluated enhanced multi-component consultation pack- 
ages that are built upon best practices in consultation (Frank 
and Kratochwill 2014) and informed by cognitive behavioral 
therapy and motivational interviewing (MI) techniques (e.g., 
Bradshaw et al. 2018; Owens et al. 2017; Reinke et al. 2008). 
MI theory purports that specific therapist strategies elicit client 
change talk, where change talk is defined as statements that 
express a desire to change, reasons or benefits of using a 
strategy, and beliefs in one’s ability to implement the plan; 
and subsequently, client change talk predicts behavior change 
or favorable outcomes (Magill et al. 2014; Miller and Rollnick 
2013). A recent review of the MI literature provides empirical 
evidence of the predictive relationship between client change 
talk and subsequent change in behaviors such as alcohol use, 
medication adherence, and healthy food intake (Romano and 
Peters 2016). 

In the context of our multi-component consultation pack- 
age, consultants assessed each teacher’s values and 
intervention-related beliefs and used MI techniques, including 
motivational rulers, to elicit change talk to address beliefs. For 
example, after the teacher and the consultant reviewed data 
from recent classroom observations, they collaboratively de- 
veloped the teacher’s plan for the next 2 weeks. Then, the 
consultant asked two motivational ruler questions: (1) 
Among all your other responsibilities how important is the 
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implementation of this plan? Importance ratings convey the 
extent to which the plan is a priority to the teacher (Miller and 
Rollnick 2013); (2) How confident are you that you can carry 
out the plan? Confidence ratings convey the extent to which 
teachers believe they can contribute to or make the change 
happen (Miller and Rollnick 2013). Consistent with MI pro- 
cedures, if the teacher’s ratings were high (9 or 10), the con- 
sultant elicited change talk by asking What makes you a [re- 
ported number] instead of a [lower number]? If the teacher 
reported a lower number (8 or below), the consultant elicited 
change talk by asking What might it take to move you from a 
[reported number] to a [higher number]? 

The results of the pilot test (Owens et al. 2017) revealed 
that teachers with lower baseline knowledge, skills, and 
intervention-supportive beliefs (i.e., more barriers to integrity) 
demonstrated more improvement in classroom management 
skills (e.g., use of labeled praise and appropriate response to 
tule violations) in response to multi-component consultation, 
as compared with a comparison condition designed to mirror 
best practices (Cohen’s d ranged from 0.33 to 1.12 across 
skills). This study and others (Bradshaw et al. 2018; Reinke 
et al. 2008) are revealing promising results about the use of MI 
in teacher consultation. 

However, we are unaware of any studies that have examined 
the relationship between teacher motivation and change in 
teacher behavior. Miller and Rollnick (2013) describe both im- 
portance and confidence as critical for motivation to change, as 
people are unlikely to commit to making a change unless they 
believe the change is important and possible. Yet, as suggested 
by MI theory, a better understanding of teachers’ ratings of 
importance and confidence could help consultants differentiate 
consultation to better support teacher needs. If a teacher had 
high importance ratings but low confidence ratings, the consul- 
tant could focus on skills practice. If a teacher had high confi- 
dence ratings but lower importance ratings, the consultant may 
focus on trying to understand how implementation could be 
better linked to other teacher priorities. Tailoring consultation 
may allow for greater facilitation of change talk, problem-solv- 
ing, and a greater reduction in barriers, ultimately leading to 
better teacher and student outcomes. 


Motivational Rulers as Assessment Tool 


Given the relationship between change talk and behavior 
change (Magill et al. 2014; Romano and Peters 2016), practi- 
tioners are interested in identifying efficient, psychometrically 
sound assessments that could provide a proxy for motivation 
for change. Motivational rulers are a common component of 
MI-based protocols and may serve as such an assessment 
(Miller and Rollnick 2013). Motivational rulers ask clients 
to rate the importance of change, readiness for change, and/ 
or confidence in change or a given plan on a | to 10 scale. 
There has been some psychometric evaluation of ruler ratings 
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in clinical contexts. Boudreaux et al. (2012) examined the 
psychometric properties of three ruler ratings (importance of, 
readiness to, and confidence in ability to quit smoking in the 
next month) in patients attending an emergency room visit. 
This study documented the moderate association between 
these constructs (7s range from 0.50 to 0.70) and the utility 
of ruler ratings in predicting smoking behavior 2 weeks later. 
However, we are unaware of any studies assessing the use of 
motivational rulers in teacher consultation. 

Thus, the goal of this study was to examine psychometric 
properties of motivational ruler ratings provided by teachers 
during the early stages of consultation in classroom manage- 
ment, as change in adult behavior is often observable after a 
brief MI intervention (Dishion et al. 2003). We sought to an- 
swer the following questions: What is the relationship be- 
tween confidence and importance (aim 1)? If ratings on these 
two variables are highly correlated, they may not represent 
distinct constructs. If the constructs are distinct, do they show 
stability (as evidence of credibility) over a brief time period 
(aim 2)? Lastly, is there preliminary evidence of predictive 
validity of ratings (aim 3)? Namely, are importance or confi- 
dence ratings associated with teacher implementation of class- 
room management strategies in the 2 weeks following the 
ratings? Findings will provide insights into the reliability 
and validity of motivational ruler ratings in this context and 
generate hypotheses for future research. 


Method 
Participants 


Participants were 29 teachers (13 in Ohio, 16 in Florida) in the 
multi-component condition of our randomized trial (Owens 
et al. 2017), as motivational ruler ratings were only collected 
in this condition. Three teachers in this condition were exclud- 
ed from this study because the target student in their class 
moved prior to the second consultation session, resulting in 
missing data for the analyses. Most (89.70%) teachers were 
women, who identified as Non-Hispanic White (51.70%) or 
Hispanic (any race; 48.30%) with an average of 13 years of 
teaching experience (range 0.5 to 36 years); 55.20% had a 
master’s degree. There were 9 consultants (see Owens et al. 
(2017) for more details about teachers and consultants). 


Procedures 


All general education teachers in the eight participating schools 
were invited to a 3-h workshop on best practices in classroom 
management and the daily report card (DRC) intervention. At the 
workshop, teachers were recruited for the consultation study 
(across schools, response rate was 10 to 58%). To participate, 
teachers were required to identify one student with or at risk for 
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attention deficit hyperactivity disorder (see target student identi- 
fication procedures in Owens et al. (2017)). Consultation focused 
on general classroom management strategies (i.e., labeled praise, 
use of rules, effective instructions, and appropriate response to 
tule violations) and the use of a DRC intervention with the target 
student. A DRC is an evidence-based classroom intervention for 
reducing disruptive behavior and improving academic perfor- 
mance. When using a DRC, teachers identify target behaviors 
(e.g., interruptions, work completion), set daily goals for success 
(e.g., 7 or fewer interruptions), provide the student with feedback 
throughout the day, and send the DRC home to parents where 
privileges can be provided contingent upon achievement of 
goals. With the consultant, teachers participated in (a) an inter- 
view about the teacher’s values and approach to classroom man- 
agement, (b) a target behavior interview to identify DRC target 
behaviors, and (c) a DRC development meeting to review base- 
line data (collected by the teacher) and finalize the DRC, each 
occurred during separate consultation meetings. Once the DRC 
was launched, teachers were observed weekly and met biweekly 
(every other week) with the consultant. During each consultation 
session (referred to as session | through 4), the consultant and the 
teacher discussed an informational handout, reviewed DRC data 
and observation data from the past 2 weeks, and created a plan 
for the next 2 weeks based on the feedback. Consultants were 
trained to adopt an MI spirit and use MI strategies during all 
sessions (see Owens et al. (2017) for details). Implementation 
integrity was conceptualized as teacher use of a strategy (e.g., 
praise) and data were obtained via observation. 


Measures 
Motivational Rulers 


At the end of each consultation session (session | through 4), 
teachers rated the importance of and their confidence in 
implementing their plan over the next 2 weeks. Ratings were 
made ona | (not at all) to 10 (very confident/important) scale. 


Student Behavior-Teacher Response (SBTR) 
Observation Rating System 


The SBTR is a psychometrically sound class-wide observa- 
tion system designed to capture discrete student-teacher inter- 
actions (Fabiano et al. 2010; Owens et al. 2017). Observers 
obtained frequency counts of (a) classroom rule violations by 
the target student, (b) classroom rule violations by all other 
students, (c) teacher’s appropriate responses to each rule vio- 
lation (by the target student and other students, separately), 
and (d) teacher’s praise statements (labeled and unlabeled) 
toward the target student and other students. Each frequency 
count variable (e.g., total rule violations, total labeled praise) 
was summed for a given observation, divided by the duration 


of the observation in minutes, and multiplied by 60 to produce 
a rate per hour. The total number of appropriate teacher re- 
sponses to target student violations for a given observation 
was divided by the total number of violations by the target 
student for that observation period. This produced the percent 
of appropriate response to violations per observation. This 
was repeated for appropriate response to violations by all other 
students. Rates and percentages were each averaged, respec- 
tively, across observations during baseline and during the 
2 weeks between consultation sessions. Inter-observer assess- 
ments were conducted for 24% of all observations. Intra-class 
correlations (ICC) of type | for average of k raters ranged from 
0.78 to 0.98 across variables. 


Results 


Aims 1 and 2: Correlations Between Importance 
and Confidence Ratings 


Motivational ruler ratings ranged from 5 to 10; with averages 
falling between 8 and 9 (see Table 1). Correlations between 
importance and confidence ratings from the same session (see 
italics in Table 1) ranged from low (session 1: r=0.24, p= 
0.22) to moderate (session 2 r=0.54, p< 0.05). Stability be- 
tween initial importance ratings at session | and later sessions 
(see boldface in Table 1) was moderate, ranging from r= 0.42 
(session 3, p<0.05) to r=0.54 (session 4, p<0.01). 
Similarly, stability between initial confidence ratings (at ses- 
sion 1) and later sessions was moderate, ranging from r = 0.32 
(session 4, p =0.13) to r=0.53 (session 3, p< 0.01). 


Aim 3: Association Between Motivational Ruler 
Ratings and Behavior Change 


Using session | importance and confidence ratings, we cate- 
gorized teachers as having lower (ratings of 5 to 8) or higher 
(ratings of 9 or 10) ratings relative to each other. These cutoff 
scores were selected because they aligned with consultant 
interview procedures (e.g., whether they elicited change talk 
in relation to a higher or lower number) and because the small 
sample size precluded a three-level or four-level group (e.g., 
representing the interaction between the two constructs). We 
examined change in six teacher behaviors between the base- 
line period and the session 2 observations. Baseline and end of 
consultation data provided elsewhere (Owens et al. 2017; 
Owens et al. 2019) document that teachers had room for 
growth in these behaviors. Cell sizes (see Table 2) vary for 
each strategy due to missing observation data (e.g., if there 
were no target student rule violations in an observation, there 
were no data for percent appropriate response). Due to small 
cell sizes, we report descriptive rather than inferential 
outcomes. 
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Table 1 

N M SD Sl- S1-I 

Cc 

S1-Confidence 28 9.18 1.19 - 0.24 
S1-Importance 29 8.10 1.45 - 
S2-Confidence 28 8.93 1.39 
S2-Importance 29 8.03 1.43 
S3-Confidence 27 9.04 1.29 
S3-Importance 27 8.26 1.58 
S4-Confidence 25 9.12 1.33 
S4-Importance 25 8.68 1.18 
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Descriptive statistics and bivariate correlations for teacher motivational ruler ratings at sessions | through 4 


$2-C $2-I $3-C $3-I S4-C S41 

0.42* 0.06 0.53** 0.27 0.32 0.17 
0.39% 0.47* 0.15 0.42* 0.04 0.54** 

- 0.54** 0.54** 0.45* 0.66** 0.48* 

- 0.20 0.62** 0.32 0.60** 

- 0.41* 0.63** 0.37 

- 0.44* 0.32 

- 0.37 


S session. Italicized type reveals the correlations between constructs within each session. Boldface type reveals the stability of the first rating with other 


ratings of its type over the next three sessions. **p < 0.01, *p <0.05 


For importance ratings, teachers with higher ratings 
showed more improvement than those with lower ratings for 
rates of praise (labeled and unlabeled) toward target student 
and other students, and percent appropriate response to other 
student rule violations (see Table 2). For these behaviors, the 
improvement of teachers in the higher rating group was nearly 
twice the rate of that of teachers in the lower rating group. 
Hedge’s g effect sizes representing group differences at ses- 
sion 2 ranged from 0.11 to 0.78. Rates of change for percent 
appropriate response to target student rule violations were 
similar for both groups. 

For confidence ratings, the pattern was similar (to impor- 
tance ratings) for praise (labeled and unlabeled) toward target 


student and other students and for percent appropriate re- 
sponse to target student rule violations. Hedge’s g effect sizes 
representing group differences at session 2 ranged from 0.20 
to 0.58. 


Discussion 


This study provides preliminary descriptive evidence of the 
psychometric properties of motivational ruler ratings with 
teachers. First, the correlations between importance and con- 
fidence rating from the same session were low to moderate. 
Consistent with Boudreaux et al. (2012), these findings 


Table 2 Rates of teacher strategy use during baseline and subsequent 2 weeks among those with high and low ratings of confidence and importance 


Importance 


Teacher strategy N Baseline, M (SD) Session 2,M (SD) Improvement ES 


Praise - TS 

Low group 18 4.78 (2.83) 4.57 (4.92) —0.21 

High group 7 4.42 (2.85) 7.20 (7.05) 2.78 
Labeled praise - TS 

Low group 18 1.85 (2.13) 3.09 (4.51) 1.24 

High group 8 1.62 (2.60) 4.03 (4.89) 2.41 
Praise - OS 

Low group 18 18.10 (9.16) 22.97 (18.36) 4.87 

High group 8 17.82 (14.58) 28.53 (25.72) 10.71 
Labeled praise - OS 

Low group 18 8.59 (7.82) 14.68 (14.18) 6.09 

High group 8 = 7.32 (7.43) 16.52 (19.09) 9.20 
% App. response to RV - TS 

Low group 16 24% (17%) 55% (78%) 31% 

High group 6 29% (25%) 59% (37%) 30% 
% App. response to RV - OS 

Low group 17 38% (16%) 47% (27%) 9% 

High group 10 41% (23%) 68% (27%) 27% 


Confidence 


N Baseline, M (SD) Session 2,M (SD) Improvement ES 


0.44 7 4.47 (2.48) 4.12 (3.82) ~0.35 0.25 
17 4.69 (3.03) 5.55 (6.27) 0.86 

0.20 7 1.11 (0.97) 1.75 (2.38) 0.64 0.45 
18 1.83 (2.44) 3.78 (5.12) 1.95 

0.27 7 14.51 (3.83) 18.89 (10.89) 4.38 0.36 
18 18.34(11.84) 26.41 (23.57) 8.07 

0.11 7 4.13 (2.68) 8.50 (5.60) 4.37 0.58 
18 8.90 (7.75) 17.36 (17.69) 8.46 

6% 4 21% (15%) 41% (27%) 20% 28% 
17 25% (20%) 61% (77%) 36% 

78% 6 38% (20%) 60% (32%) 22% 20% 
20 39% (19%) 54% (29%) 15% 


TS target student; OS other students; App. appropriate; RV rule violations; ES Hedge’s g effect size representing group differences at session 2 
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suggest that ratings of importance and confidence are related 
yet distinct. Thus, when a teacher’s importance and confi- 
dence ratings differ, it may indicate that different strategies 
should be used in consultation to facilitate implementation 
(e.g., additional skills practice, or exploring the connection 
between teacher values, priorities, and recommended prac- 
tices). Researchers and practitioners should continue to exam- 
ine importance and confidence ratings as separate constructs. 

Second, we found moderate stability between initial impor- 
tance ratings and importance ratings over the first several ses- 
sions, and between initial confidence ratings and confidence 
ratings over the first several sessions. This offers some credi- 
bility to the ratings in this context (i.e., ratings were not hap- 
hazard), as only moderate stability is expected knowing that 
perceived importance and confidence may change as teachers 
observe change in student behavior and receive feedback on 
their own behavior. 

Lastly, we found evidence that motivational ruler ratings 
may have utility in predicting change in teacher behaviors 
during the early phase of consultation. Namely, teachers 
who were highly confident in session | that they could com- 
plete their plan nearly doubled their use of five of the six 
recommended strategies between baseline and session 2. 
Although teachers with lower confidence showed some i1m- 
provement in these six behaviors, the rate of improvement was 
approximately half that of those with high confidence ratings. 
The pattern was similar for importance ratings. Thus, despite 
teachers only using ratings from 5 to 10 (on a | to 10 scale) 
and despite the modest correlations between importance and 
confidence, both importance and confidence ratings were as- 
sociated with short-cycle changes in several teacher 
behaviors. 

Although this study makes a unique contribution to the 
literature, the findings should be considered preliminary given 
the limitations of the sample. First, the sample is small, and the 
categorical groups of teachers are even smaller. This preclud- 
ed statistical analyses and examination of each combination of 
high and low levels of both constructs (importance and con- 
fidence). Thus, the findings warrant replication with larger 
samples to ensure stability. Second, we categorized teachers 
with ratings of 9 or 10 into the higher group and teachers with 
ratings of 5 through 8 in the lower group. Although this cate- 
gorization aligned conceptually with the clinical practice (e.g., 
how consultants used the ratings to elicit change talk), differ- 
ent categorizations should be considered. Third, other vari- 
ables (student behavior, teacher characteristics, contextual fac- 
tors) were not accounted for and warrant investigation given 
the complexity of factors affecting teacher motivation and 
behavior. Fourth, the range of the motivational ruler ratings 
was restricted and skewed towards the upper end of the scale. 
Additional studies are needed to determine if teachers do not 
use the low end of the scale or if this range is specific to this 
sample. 


Nonetheless, this study provides the first examination of 
the psychometric properties of motivational ruler ratings com- 
pleted by teachers in the context of consultation focused on 
classroom management interventions. The findings suggest 
there is adequate variability and stability in ratings, and prom- 
ise for the short-term predictive validity of the ratings. Thus, 
researchers should further explore these brief tools as a proxy 
for motivation for change using outcomes at a variety of time 
frames and with a larger sample. With replication, such ratings 
could be used by consultants to individualize consultation to 
teacher needs and as a way to enhance teacher motivation and 
implementation, and ultimately, student outcomes. 
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